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1. INTRODUCTION 


1.1. Background 

Digital computers are having an increasingly important role in process control appli- 
cations, particularly those in which human life may be endangered such as space vehicle 
and avionic systems, C 3 I systems, and medical life-support systems. These systems 
have stringent reliability requirements as system failure is potentially life-threatening. 
An example is the working figure of 10~ 9 for a ten hour flight being used as a require- 
ment for system failure probability by NASA-Langley Research Center (NASA-LaRC). 3 

The need for predicting the reliability of the software components of these life- 
critical systems has become more apparent with the increasing functionality being 
ascribed to them. The absence of a credible reliability prediction methodology for highly 
reliable software makes the system reliability analyses chimerical at best. The develop- 
ment of this methodology is hindered by our lack of knowledge about the underlying 
nature of the failure process for embedded real-time control software. This lack of 
knowledge contributes to our inability to completely eradicate or tolerate faults and to 
our lack of confidence in the extent to which we have approximated the goal of zero 
defects. As evidenced by the first well-publicized Space Shuttle software bug, the failure 
of the initialization logic in J. Garman’s words resulted from a “very small, very improb- 
able, very intricate, and a very old mistake." 2 

This bug typifies the rare and convoluted combination of events which cause care- 
fully developed software to fail. It is this type of residual fault which surfaces infre- 
quently causing a rare event or small probability failure. To detect these faults requires 
an order of magnitude longer time under test than the target mean time to failure. 3 

To contribute to the development of a sound statistical methodology for estimating 
software reliability, radar tracking software was tested using the repetitive run approach 
for fault rate estimation and n- version programming for error detection. 4 Four imple- 
mentations of a Launch Interceptor Condition (LIC) module for a radar tracking appli- 
cation have been subjected to a long time under test with over 15,000,000 test cases 
being executed. To expedite the collection of repetitive run failure data, the AUTOSIM 
tool was developed. 

1.2. Repetitive Run Testing 

Repetitive run testing was first advocated by Nagel and Skrivan of Boeing Com- 
puter Services. 5 The repetitive run test approach provides information about the proba- 
bilistic impact of detected software faults on subsequent fault detection. It involves 
repetitively executing a software program using different sets of test cases from its initial 
state or design stage (usually the code version at the end of acceptance testing) through 
to the detection and correction of m faults. A code version is an instantiation of an 
implementation of the code under test. During the repetitive runs, the sequence of pro- 
gram fixes result in several instantiations or versions of the code. 

Repetitive run testing provides “better" estimates of the individual fault rates. On 
subsequent runs or replications, the testing begins again with the initial version of the 
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code under test, and the faults are corrected again. Replications continue until enough 
observations have been collected to achieve the desired level of statistical accuracy for 
estimating the program failure rates. Since the input stream of test data differs for each 
replication, the order in which faults are diagnosed and the correction applied also 
differs for each replication. 

1.3. N-Version Error Detection 

To detect output errors from the radar tracking software, the technique of n-version 
programming was employed. N-version progr ammin g, first widely publicized by 
Avizienis 6 and further discussed by Anderson and Lee , 7 involves n programmers 
independently coding the problem from the same specification. A software tool, the N- 
VERSION CONTROLLER , 8 ’ 9 which controls the execution of each of the n indepen- 
dently coded implementations of the code under test and signals discrepancies between 
the n output vectors, was constructed for this purpose. The codes under test are the 
software modules being tested for their reliability. We refer to each code under test as 
an Application Task; (ATj) where i=l,...,n. Using n-version programming for error 
detection avoids reliance on a standard to determine output correctness. 

1.4. The Need for AUTOSIM 

The need for a system to automate the repetitive run test process became apparent 
during the testing of the radar tracking software once we observed that diagnosing the 
fault and correcting the code under test was a time consuming, error prone process. Per- 
forming the diagnosis-correction task requires an individual with at least one year of 
programming experience. However, after repeating the task for several replications, it 
becomes very mundane and the programmer begins to perform this task by rote making 
it a tedious, error prone process. Moreover, the timely completion of the diagnosis- 
correction task is contingent upon the availability of the programmer. Since the programs 
fail at random points in time, the speed with which the data is collected is inhibited by 
the programmer’s availability. For these reasons, we decided to develop the AUTOSIM 
system which performs repetitive run testing with a minimum of human intervention. 

1.5. Definition of Terms Related to AUTOSIM 

Understanding nomenclature throughout the AUTOSIM report is essential to under- 
standing the purpose and functionality of the AUTOSIM tool. We use the following 
terms throughout the report. The definitions of failure, error, and fault are consistent 
with those defined in “Fault Tolerance by Design Diversity: Concepts and Experi- 
ments .” 10 

CODES UNDER TEST — The software modules being tested for their reliability which 
are referred to as AT; for Application Task; . 

CONDITIONS MET MATRIX (CMM) — The principal output from the Launch Inter- 
ceptor Condition (LIC) Application Tasks. 
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CMS — VAX 11/780 Code Management System 


DESIGN STAGE — Versions of the code under test during repetitive run testing 

ERROR — The discrepancies noted (i.e., the incorrect element(s) of the output vari- 
ables) 

FAILURE — THE N- VERSION CONTROLLER signals a discrepancy in the output 
variables of the launch interceptor condition software. For this problem, an application 
task fails when it incorrectly disagrees with any of the other application tasks or the 
extensively tested version. 

FIX/FAULT — A FIX is the minimum code change required to correct a single error. 
The FAULT is the conceptual flaw in the software which is corrected by a fix and is the 
cause of the error. For simplicity, we consider the fault to be defined by the fix and use 
the term fault in lieu of the term fix throughout the report although we recognize the 
real fault is not uniquely defined. 

IMPLEMENTATION — An independently coded version of the same functional specifi- 
cation (i.e., one of the n-versions of the code under test) 

LAUNCH — Critical output variable from the LIC code under test. 

LAUNCH INTERCEPTOR CONDITION (LIC) Problem — A radar tracking applica- 
tion which is the first AUTOSIM test specimen. 

N-VERSION CONTROLLER — Testbed used for testing n versions of an AT in paral- 
lel. 

N-VERSION CONTROLLER INTERFACE — Tool used for monitoring the execution 
of the N-VERSION CONTROLLER and viewing stored test results. 

REPLICAnON/REPEilTlVE RUN — Repeats of the repetitive run test be ginnin g 
with the initial version of the code under test and using a different set of test cases. A 
REPLICATION is sometimes referred to as a REPETITIVE RUN. 

VERSION — An instantiation of an implementation of the code under test. During the 
software fault diagnosis-correction process, the program fixes result in several instantia- 
tions or versions of the code. 
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2.0. OVERVIEW OF THE AUTOSIM TOOL 

2.1. Design Goals 

The automation of the error diagnosis-correction tasks requires a knowledge base 
that contains information about the documented software faults and the associated code 
fixes, as well as knowledge about the fault diagnosis process. These requirements 
categorize the AUTOSIM system as an “expert" system which replaces a low level of 
programming expertise. 

In designing the AUTOSIM system, we pursued the following goals: 

• separate concerns by maintaining a clean interface with the N- VERSION CON- 
TROLLER and isolating the information specific to the code under test 

• make the system general by storing the information specific to the code under test 
and the error detection state information in generalized data structures 

• minimiz e the complexity of the fault diagnostic task by defining error classes, 
which are handled separately, and developing an efficient algorithm to minimiz e 
the fault diagnostic time. 

Accomplishment of the first goal is important if AUTOSIM is to be used with test tools 
other than the N- VERSION CONTROLLER and with different application code under 
test. The second goal minimizes the amount of overhead required when modifying the 
system to test code from other applications and other error detection tools. The 
existence of non-unique 1-to-n mappings of faults to errors necessitates achievement of 
the third goal. 

2.2. The AUTOSIM Algorithm 

AUTOSIM performs two principal functions: fault diagnosis and fault correction. 
Fault diagnosis entails identifying which fixes to apply to the failed code based on the 
information contained in the N-VERSION CONTROLLER state vector. Critical to this 
identification is the implementation of an efficient algorithm for rapid identification and 
testing of the candidate fixes. Fault correction entails inst allin g the appropriate fix and 
resuming the n-version testing. The logic for fault correction is similar to the logic for 
the testing of candidate fixes. Figure 1 provides a pseudo-code description of the 
AUTOSIM algorithm. 
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ON STOP OF TEST 
FETCH TEST STATE 
DETERMINE FAILED CODE(S) 

FOR EACH FAILED CODE 

TEST CODE VERSION WITH ALL KNOWN FAULTS CORRECTED 
ON FAILURE SIGNAL USER 

DETERMINE ERROR(S) 

FOR EACH ERROR 

DETERMINE ERROR CLASS 

IF CLASS- ADDRESS 

DETERMINE CANDIDATE ADDRESS FIXES 
FOR EACH ADDRESS FIX 

INSTALL AND TEST ADDRESS FIX 
ON SUCCESS GET NEXT ERROR 
CALL USER 


IF CLASS = ABEND 

DETERMINE CANDIDATE ABEND FIXES 
FOR EACH ABEND FIX 

INSTALL AND TEST ABEND FIX 
ON SUCCESS GET NEXT ERROR 
CALL USER 

IF CLASS= ERROR 

DETERMINE CANDIDATE ERROR FIXES 
FOR EACH ERROR FIX 

INSTALL AND TEST ERROR FIX 
ON SUCCESS GET NEXT ERROR 
CALL USER 
RESUME TEST 


Figure 1. 

Pseudo-Code Description of 
Fault Diagnosis- Correction Algorithm 



2.3. The AUTOSIM Design 

Figure 2 provides a structural view of the AUTOSIM tool. The diagram shows the 
quasi-static data structures, which remain relatively constant during testing, and the 
dynamic data structures, which are updated by either the AUTOSIM software or the N- 
VERSION CONTROLLER software. 

The contents of the quasi-static data structures depend on the code under test. The 
overwrite, abend, and output error maps contain information on which code fixes are 
associated with different types of faults. These structures are quasi-static because they 
are updated only when a new fault is identified. This identification results from the 
human intervention required when AUTOSIM fails to diagnose the fault (i.e., there are 
no valid entries in the error maps described in Section 4.2 of this report). 



Figure 2. 

A Structural View of the AUTOSIM Tool 
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3. IMPLEMENTATION OF THE AUTOSIM TOOL 
3.1. Control Flow 

Figure 3 provides a high level description of the AUTOSIM control flow and the 
associated data files. The three major functional modules which comprise AUTOSIM 
are FIXID, MEASJMP, and RESTORE. The VMS co mm and procedures invoked by 
AUTOSIM (also depicted in Figure 3) are SIMBATCH.COM, FTXAPP.COM, and 
VTEST.COM. There are 7 major types of files used by AUTOSIM. These files are 
ATGEN.DAT, EXECUTION.DAT, iABEND.DAT, iCLOBBER.DAT, iCMM.DAT, 
iLAUNCH.DAT, and ERRORS.DAT. 

The following sections describe the AUTOSIM functions, files, and command pro- 
cedures. 
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INPUT 

FILES 


FUNCTION 


OUTPUT 

FILES 


ATi 

Mill roc code 
STATUS. DAT 
SIM.DAT 


EXECUTION.DAT 

ATGEN.DAT 

iCLASS.DAT 

PROBli.FOR 
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nil 


SIM.DAT 


goto 1 with 
ATGEN.DAT 
iCLASS.DAT 
STATUS.DAT 


iCLASS.DAT 


ATGEN.DAT 
invokes CMS commands 


PROBli.FOR 

compiles, links, executes 
Version Tester 

EXECUTION.DAT 


EXECUTION.DAT 
CMM_FILE(i) • 
ATGEN.DAT 


improvement 


i 


restore 


iCLASS.DAT 

goto 4 

ATGEN.DAT 


Figure 3. 

AUTOSIM Global Control Flow 





3.2. Descriptions of AUTOSIM Functions 

The following alphabetical list provides brief descriptions of all AUTOSIM func- 
tions. Schematic logic diagrams which describe the interfaces between these functions 
are provided in Appendix A. 

ABENDJERROR — Fmds the CMS generation for AT; abend error. 

APPEND — Adds fixes into the fix-list for ATj. 

AUTOSIM — Keeps the simulator running by identifying errors and installing fixes in 
the individual applications tasks. 

B_SEARCH — Finds the fixes associated with the last element generations that were 
superseded into AT ; ’s CMS class. 

CLOBBER^ERROR — Finds the CMS element generation to fix ATj overwrite. 

CLOSE _FTLES — Closes AUTOSIM data files before spawning a subprocess. 

CMM_ERROR — Finds the CMS element generation with a fix for a particular AT; 
output error. 

ECHOJERROR — Writes AUTOSIM error diagnostics to file AUTOERR.DAT. 

FIXED — Identifies where an application task failure occurred and determines which ele- 
ment generation to supersede in the CMS class. 

GET_SIMJDATA — Gets a snapshot of global memory and prepares for new design 
stage or new replication. 

EDJFTXES — Identifies the fixes associated with any CMS element generation. 

L AUN CH_ERROR — Fmds the CMS element generation for ATj launch error. 

MAKE_ICLASS — Builds the file which communicates with the DCL command pro- 
cedure FIXAPP.COM. The file iCLASS.DAT tells FTXAPP.COM what elements are to 
be superseded by new generations into the appropriate CMS class. 

MEASURE_IMP — Measures the effect that the last fix had on a particular AT;'s 
CMM. 
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OPEN_FTLES — Opens the global input & output files. 

PREP_GLO — Prepares global memory for either a new design stage or a new replica- 
tion and builds the command procedure FORLINREP.COM that will compile the modi- 
fied application tasks. Places the object module in the library SIMDISK:PROBILIB.OLB 
and links a new executable image of the N- VERSION CONTROLLER. 

READ _ATGEN — Reads AT; records from ATGEN.DAT. 

READ_CMMFILE — Reads the available fixes contained in CMS element generations 
from the iCMM.DAT file for the AT; under consideration. 

READ_EXECUTION — Reads the file ASIM_DATA: which serves as the communica- 
tion link between AUTOSIM and the spawned subprocess executing VTEST.COM. 

RE AD_SIM. DATA — Reads last record written to SIM.DAT. 

READ_TRACEBACK — Reads the AT; traceback from ERRORS.DAT after an abend 
has occurred. 

RESTORE — This module is executed in the event that a lame element generation was 
superseded into the ATj’s CMS library, i.e., the element generation that was last 
inserted did not contain the necessary fix. Restores the last element generation that was 
superseded into the ATj’s class to its previous generation and find the next element gen- 
eration to install. 

SET_MASK — Sets the fix mask which is displayed by the INTERFACE to the N- 
VERSION CONTROLLER and allows the operator to know what fixes are currently 
installed. 

SPY_GLOBAL — Accesses global memory and returns latest sequence number. 

VTEST — Tests an individual AT with the contents of the N-VERSION CON- 
TROLLER INPUT.DAT file and compares the CMM and LAUNCH output to that 
computed by the golden AT. 

VOID_FDC_LIST — Initializes the fix list to zero. 

WRITE_ATGEN — Updates application task records in the file 
ASIM_DATA: ATGEN.DAT. ATGEN.DAT is a direct access file containing 
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information about the CMS class for each application task. 

WRITE.EXECUnON — Writes to the file ASIM_DATA:EXECUTION.DAT. 

ZERO_SUPER_ELE — Initializes the supersede section (the second group of N_ATS 
records) in the file ATGEN.DAT. This is to ensure that only the next 
element/generation elected to fix an error will be superseded into the appropriate AT; 
class in the CMS library. 


3.3. Descriptions of AUTOSIM Files 

The following alphabetical list provides descriptions of the AUTOSIM data files. 
More complete descriptions can be found in Appendix B. 

ATGEN.DAT — Contains the trace of CMS element generations that were installed, 
indicates those that are presently installed, and those which can be superseded. 
ATGEN.DAT is accessed by the functions READ_ATGEN and WRITE _ATGEN. 

EXECUTION.DAT — Contains information pertaining to results of executing the code 
in error with the diagnosed fix. EXECUTION.DAT is accessed by the functions 
READ_EXECUTION and WRITE_EXECUTION. 

LABEND.DAT — Contains a history of the abend failures which have been documented 
for each AT, during the execution of the Launch Interceptor Code. The function 
ABEND _ERROR accesses LABEND.DAT. 

iCLOBBER.DAT — Contains a history of the overwrite failures which have been docu- 
mented for each AT; during the initial execution of the Launch Interceptor Code. The 
function CLOBBERJERROR accesses iCLOBBER.DAT. 

iCMM.DAT — Contain the fault-error maps for the documented faults. The function 
CMM_ERROR accesses iCLOBBER.DAT. 

iLAUNCH.DAT — are used when an AT; fails and the failure does not show up as an 
abend, overwrite or as disagreement in the Conditions met matrix. The function 
LAUNCH_ERROR accesses iLAUNCH.DAT. 

ERRORS.DAT — Contains all system trackbacks when a routine in the N-VERSION 
CONTROLLER ends abnormally. The function READ_TRACEBACK accesses 
ERRORS.DAT. 
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3.4. AUTO SIM Command Procedures 

The following alphabetical list provides brief descriptions of the AUTOSIM com- 
mand procedures. Listings of the command procedures can be found in Appendix C. 

AUTOSIM.COM — Submitted as a batch job to start the execution of AUTOSIM. 

FDCAPP.COM — Creates working versions of each of the ATs. 

FORLINREP.COM — A dynamically created command procedure which compiles the 
application tasks, places them in PROB1LEB.OLB (the problem 1 object library) and 
links the N- VERSION CONTROLLER. 

SIMBATCH.COM — Submitted as a batch job to start the execution of the N- 
VERSION CONTROLLER. 

VTEST.COM — Tests fixed version of AT with error-invoking input case. 
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4. AUTOSIM DEPENDENCIES 


4.1. Management of the Code under Test 

The VAX1 1/780 Code Management System (CMS) is used to manage the library 
containing the code under test. 11 CMS is a program library system for software develop- 
ment and maintenance which operates as an online librarian by keeping track of the 
source code files. Each CMS library contains elements, generations, and classes. 

A CMS element is the basic structural unit in a code library. An element consists of 
one or more ASCII files that represent a meaningful unit. An element is created and 
named when a file ( or files ) is transferred from a working directory to the CMS library 
via the CMS CREATE ELEMENT co mm and. The AUTOSIM elements are functionally 
defined modules of the each implementation of the code under test. 

A CMS element generation represents a phase in the development of that element. 
When an element is created and placed in the CMS library for the first time, CMS 
creates generation one of that element. Each time the element is reserved, modified, and 
replaced in the CMS library, a new generation is created. The AUTOSIM generations 
are the functionally defined modules with different fixes applied. 

A CMS class is a set of generations of different elements. A class is established to 
define a set of generations that make up the whole of part of a software system at a 
specific stage of development. The AUTOSIM classes pertain to the different codes 
under test, i.e., to the different ATj . 

Tables 1, 2, and 3 depict the organization of the code library for the testing of the 
Launch Interceptor Condition Software. The module names are the actual names of the 
different software modules given by the progr amm ers. A number identifies each fault in 
an ATj. 

In developing the AUTOSIM code library, we partitioned each AT into modules 
which correspond to conditions of the LIC problem and into modules which contain sub- 
routines common to the test condition modules. Modules with no fixes were coupled 
with modules that had corresponding fixes (i.e., placed in the same CMS element) to 
save storage space. Subsequent generations of each element are the versions of the code 
under test with different fixes applied. CMS stores the subsequent generations of a 
CMS element by retaining the code differences from the first generation element. The 
update of the version of the code under test to correct a fault does not necessarily result 
in the installation of the next generation of an element. For example, installation of the 
fix associated with Fault 8 may be required for element A09 of ATI prior to installation 
of the fix associated with Fault 7. 


4.2. Code Under Test Fix-Error Maps 

Fix-error maps define the relationship between code under test output errors (in this 
case errors in the CMM and in LAUNCH) and fixes which correct those errors. Tables 
4, 5 and 6 depict the fix-error maps for the three implementations of the LIC problem 


13 



TABLE 1. CLASS DESCRIPTIONS OF AT; 


CLASS 

ELEMENT 

GENERATION 

DESC1 

OTTION 




Module Name 

Faults Corrected 

ATI 

ADI 

1 

main 




2 

main 

12 


AD2 

1 

condl 




2 

condl 

9 


AD3 

1 

cond2 

cond3 




2 

cond2 

cond3 

10 


AD4 

1 

cond4 

cond5 




2 

cond4 

cond5 

2 



3 

cond4 

cond5 

4 



4 

cond4 

cond5 

2,4 


AD5 

1 

cond6 
cond7 . 




2 

cond6 

cond7 

3 


AD6 

1 

cond8 




2 

cond8 

6 


AD7 

1 

cond9 

condlO 




2 

cond9 

condlO 

11 


A08 

1 

condl 1 





condl2 

condl3 

condl4 

condl5 





main 



AD9 

1 

anglea 




2 

anglea 

1 



3 

anglea 

1,7 



4 

anglea 

1,8 



5 

anglea 

1,7,8 


AD10 

1 

dista 





rad 




2 

dista 





rad 

5 












TABLE 2. CLASS DESCRIPTIONS OF AT; 


CLASS 

ELEMENT 

GENERATION 

DESCI 

Module Name 

UPTTON 
Faults Corrected 

AT2 

B01 

1 

2 

probib 

probib 

1 










TABLE 3. CLASS DESCRIPTIONS OF AT; 


CLASS 

ELEMENT 

GENERATION 

DESCRIPTION 




Module Name 

Faults Corrected 

AT3 

C01 

1 

main 





condl 





cond2 




2 

main 





condl 



. 


cond2 

3 


02 

1 

cond3 




2 

cond3 

4 


C03 

1 

cond4 




2 

cond4 

5 


C04 

1 

cond5 




2 

condS 

6 


C05 

1 

cond6 





cond7 




2 

cond6 





cond7 

1 



3 

cond6 





cond7 

1.7 



4 

cond6 





cond7 

1,16 



5 

cond6 





cond7 

1,7,16 


C06 

1 

cond8 




2 

cond8 

8 


C07 

1 

cond9 




2 

cond9 

9 


C08 

1 

condlO 




2 

condlO 

10 


C09 

i 

condl 1 




2 

condll 

11 


C010 

1 

condl2 




2 

condl2 

12 


CB11 

1 

condl3 




2 

condl3 

2 



3 

condl3 

13 



4 

condl3 

2,13 


012 

1 

condl4 




2 

condl4 

14 


013 

I 

condl5 




2 

condl5 

15 


014 

1 

radcir 




2 

radrir 

18 


015 

i 

perdis 




2 

perdis 

20 


016 

1 

aglcos 




2 

aslcos 

17 



3 

aclcos 

17,19 


CO 17 

1 

abdsa 





ordnat 





disc 





errstp 





vfind 





aretri 





quad 





verin 





verout 






respectively. The rows in these tables identify the fault/fix number; the columns in these 
tables identify bit errors in the CMM, LAUNCH, abend errors and overwrite errors. 

4.3. N- VERSION CONTROLLER Dependencies 

The functions PREP_GLO and GET_SIM_DATA comprise the two points of com- 
munication of AUTOSIM with the N-VERSION CONTROLLER. 

PREP_GLO prepares global memory for either a new design stage or a new replication, 
and builds the command procedure FORLINREP.COM that will compile the modified 
application tasks. It then places the object module in the library 
SIMDISK:PROBILIB.OLB and links a new executable image of the N-VERSION 

TABLE 4. FIX-ERROR MAP FOR ATI 


FIX 

CMM OUTPUT (i) 

ABENDS 

OVERWRITES 

ID 

1 

2 

3 

D 

5 

6 

n 

8 

9 

10 

ii 

12 

13 

14 

15 

RAD 

ANGLEA 

COMMON2 

1 


















X 

2 





B 














3 







B 












4 





B 














5 
















X 



6 








B 





* 






7 



B 







B 



B 




X 


8 

















X 


9 

m 


















10 



B 
















11 










B 









RW 










B 










X indicates an error in output CMM (i), an ABEND error, or an OVERWRITE error. 


* indicates that FIX 6 is applied when simultaneous errors in CMM 8 and 13 occur. 

TABLE 5. FIX-ERROR MAP FOR AT2 


FIX 

C 

MM OUT 

PUT 

31 

ID 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

LAUNCH 

1 
















X 


X indicates an error in output CMM (i), an ABEND error, or an OVERWRITE error. 
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X indicates an error in output CMM (i), an ABEND error, or an OVERWRITE error. 


FIX 3 and FIX 4 constitute changes to perceived faults which were non-existent. 
CONTROLLER. 

GET_SIM_DATA, upon normal halting of the N-VERSION CONTROLLER, reads the 
record corresponding to the last sequence number from the SIM.DAT file. If the halt 
occurs during a replication, the module FDGD is invoked. If the halt occurs at the end 
of a replication, the command procedure FIXAPP.COM is spawned. 

4.4. VMS Dependencies 

The VMS command procedures VTEST.COM, SIMBATCH.COM, 
AUTOSIM.COM, and FORLINREP.COM are critical to the execution of AUTOSIM as 
described in Section 3.4. These procedures may change as a result of upgrading of the 
VMS DEC Command Language.* 2 


18 





















5. AUTOSIM VALIDATION AND PERFORMANCE 


5.1. Validation Test Results 

Upon completion of unit testing of the AUTOSIM modules and white box functional 
testing of AUTOSIM, we tested AUTOSIM by repeating eight of the repetitive runs or 
replications executed during a previous conduct of the experiment. 4 The test replications 
selected were replications for which interacting faults were and were not observed. The 
testing surfaced the following problems: 

• AUTOSIM execution of replication 1 identified a fix that should not have been 
applied. 

• AUTOSIM execution of Replication 2 surfaced an error in the subroutine 
GENER of the N- VERSION CONTROLLER where an input (x,y) pair was not 
re-initialized. 

• AUTOSIM execution of Replication 16 indicated that in the original replication a 
program version being tested had a fix to a previously corrected error inadver- 
tently removed. 

Logs describing the validation testing are provided in Appendix D. 

5.2. Performance Measures 

Performance of AUTOSIM was of interest for two reasons. First, we were 
interested in the reduction in human effort actually achieved from using the AUTOSIM 
tool. Replications of length 10,000 input cases executed without the use of AUTOSIM 
averaged two days to complete. These replications required the availability of a pro- 
grammer who spent approximately four hours per day monitoring the system and instal- 
ling the appropriate fixes. The AUTOSIM replications of the same length averaged six 
calendar hours to complete and require a maximum of one-half hour of monitoring time 
per day. 

Second, we are interested in the efficiency of the AUTOSIM algorithm in diagnos- 
ing the appropriate fault. We are currently measuring the efficiency by number of 
VTEST invocations per fix required. The LOG. DAT and the CHART.DAT files con- 
tain the data required to compute this statistic. 
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6. USING AUTOSIM 


6.1. Getting Started 

AUTOSIM is executed through the N- VERSION CONTROLLER INTERFACE. ^ 
To start AUTOSIM, type SIM1, select option 13, and answer the prompts. 

6.2. Libraries Needed 

The following libraries are needed to run AUTOSIM: 

PROB1LD3.TLB — which contains the N- VERSION CONTROLLER and the AT source 
code. 

AUTO.TLB — contains the AUTOSIM source code. 

MTRSRC.TLB — contains the N- VERSION INTERFACE source code. 

These files presently reside on AIRLAB System 3::DRAO:[SIM.PROB1...]. 

6.3. AUTOSIM Error Files 

The AUTOERR.DAT file contains AUTOSIM error information. Two types of 
information are stored in this file. The VMS System error message number when a sys- 
tem level error occurs and an AUTOSIM error message when AUTOSIM cannot nor- 
mally execute its function. The former system level messages can be obtained by typing 
“exit [msg. no.]". The latter AUTOSIM error messages should be completely self- 
explanatory. 
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APPENDIX A. AUTOSIM SCHEMATIC LOGIC DIAGRAMS 



Each of the following pages corresponds to a single AUTOSIM module, or to part 
of an AUTOSIM module. The module name is given in the upper left hand corner of 
each page, and other AUTOSIM routines called by the module or module fragment are 
listed in the lower left part of the page. Modules are numbered sequentially in order of 
occurrence in the drawings, from first drawing to last, and in left- to- right order within 
each drawing. Arrows drawn from a subprocess box point to the number of a called 
module or to a continuation page for the current module. Continuation pages are num- 
bered as fractional extensions of the module number; e.g., module n. would have its 
continuation pages numbered n.l, n.2, and so on, and continuation page n.m would 
have its sub-continuation pages numbered n.m.l, n.m.2, etc. 

Each box on the diagrams represents a process or subprocess in the AUTOSIM, and 
lines connecting the boxes represent program control flow. Subprocess execution is from 
top of page to bottom, and from left to right within the same level. Conditionally exe- 
cuted boxes have a small circle drawn in the upper right hand comer, and the branch 
condition is printed above the box. Boxes that are executed iteratively have an an aster- 
isk drawn in the upper right hand comer, and the iteration condition is printed above 
the process box that controls the iterated subprocess. Null branches are indicated by a 
horizontal line above the word “Null.” 

Data assignments are listed below a process box, if they’re needed to understand 
process logic. An escape on an error condition is indicated by an arrow leading from an 
empty process box. Section 3.2 of this document contains an alphabetized list of all 
modules, with brief descriptions. 

Jackson, 13 pp. 17-42 describes the notation used in these diagrams in greater detail. 
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module: B_SEARCH 



externals: 


to 

-o 


none 
















module: CLOBM-RJiRROR 



Externals: 


ECHO_ERROR 






module: CLOSE_FILES 


""i 




function 
CLOSE_ FILES 


close (unit=l, SIMDISK:SIMDAT) 
close (unit =3, ATGEN.DAT) 
close (unit =4, EXECUTION.DAT) 
close (unit=10, AUTOERR.DAT) 
close (unit=ll, 1CLASS.DAT) 
close (unit=12, 2CLASS.DAT) 
close (unit =13, 3CLASS.DAT) 
close (unit =15, FORLINREP.COM) 


Externals: 

none 
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call 
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externals: 
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Externals: 
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module: MEAS_IMP 
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READ_EXECUTION 
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module: OPEN_FTLES 


function 

OPEN_FTLES 


open (unit= 1, SIMDISK; SEMDAT) 
open (unit=3, ATGEN.DAT) 
open (unit=4, EXECUTION.DAT) 
open (unit =10, AUTOERR.DAT) 
open (unit=ll, 1CLASS.DAT) 
open (unit=12, 2CLASS.DAT) 
open (unit =13, 3CLASS.DAT) 
open (unit =15, FORUNREP.COM) 


Externals: 

ECHO.ERROR 
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module: PREP.GLO 


function 

PREP_GLO 


r.toppcr_flag= .false. 
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call 

SET_MASK 
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APPENDIX B: AUTOSIM FILE DESCRIPTIONS 



ATGEN.DAT: 


purpose: 

Keep track of what CMS element generations are presently installed in the 
ati_class and what CMS element generations were previously installed in 
the ati_class. 

background: 

Each of application task has its own CMS class, i.e. there are N_ATS 
CMS classes defined within the CMS library ASIM_CMS_LIB. There are 
N_ELE(i) elements in the CMS class corresponding to ATi. N_ELE is an 
N_ATS integer*4 array defined in AUTO. INC along with other data variables. 
ATGEN.DAT is organized into three different groups of records; each group 
has N_ATS records. All application tasks have one record within group of 
N_ATS records. Specifically, ATi has its records located at record 
positions i, i+N_ATS, and i+(2*N_ATS). NOTE! To enable this random access 
of records, ATGEN.DAT was created as a DIRECT access file. To insure 
proper file integrity DO NOT EDIT ATGEN.DAT. Records within the file 
ATGEN.DAT may be modified manually with the Fortran program MODATGEN. 

reference: 

ATGEN.DAT is referenced with unit number 3 in the open, read, and write 
statements to this file, the open statement is as follows. 

open( unit=3, file=ATGEN.DAT, access=’DIRECT’, status=’OLD’ ) 

data structures: 

at_ele - a 17 by N_ATS element byte array containing the fisrt group 
of N_ATS records. This group of records indicates what 
generations of the CMS elements are currently installed 
in the CMS class for each application task. 

super_ele - a 17 by N_ATS element byte array containing the second group 
of N_ATS records. This group of records indicates what CMS 
class elements are to be superseded with new generations. 

restore_ele - a 17 by N_ATS element byte array containing the third group 
of N_ATS records. This group of records indicates what 
generations of the CMS elements were installed in the previous 
version of the CMS class for each application task. 

record formats: 

first group: at._ele -or- 
atl’s (10i3) 
at‘2’s (01i3) 
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at3’s (17i3) 

second group: super_ele -or- 
atl’s (10i3) 

at2’s (01i3) 

at3’s (17i3) 

third group: restore_ele -or- 
atl’s (10i3) 

at2’s (01i3) 

at3’s (17i3) 



AUTOERR.DAT: 


purpose: 

log AUTOSIM errors of the nature (1) out fixes for an ATi, or (2) open, 
read, or write errors. 

reference: 

AUTOERR.DAT is referenced with unit number 10 in the following open 
statement. 

open(unit=10, file=’AUTOERR.DAT’, access=’SEQUENTIAL’, status=’OLD’) 
data structures: 

module - character*(*), the module in which the error occurred 

file - character*(*), the file that was be referenced when the 
error occured 

fail_msg - the message accompanying the failure (i.e. open, read,...) 

iostat - i*4, either the return status code from a open, read,... , or 
the CMM that did not have any more fixes 

stamp - character*23, the time the failure occurred 

record format: 
stamp 

-or- (a) Itime in format dd-mmm-yyyy hh:mm:ss.cc 
module file fail_msg iostat 

-or- (’’ERROR: <’, a, ’> ’, a, lx, a, iostat = ’, ilO) 
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EXECUTION.DAT: 


purpose: 

serves as the communication link between the spawned subprocess VTEST 
and the AUTOSIM program. VTEST updates the last two records w-hich 
contain the output from the GOLD and the ATi’s CMMs. MEAS_IMP, a function 
of AUTOSIM, updates the first record which contains the number of 
differences between the GOLD’S and the ATi’s CMMs. 

reference: 

EXECUTION.DAT is referenced as unit 4 in the following statement: 

open(unit=4, file= ’EXECUTION.DAT’, aecess=’SEQUENTIAL’, status=’OLD’) 
data structures: 

rem_n_diff - a three element integer*4 array containing the number of 

differences between the GOLD’S CMMs and a particular ATi’s. 
the valid range of values for these fields is a integer 
greater than or equal to 0 but less than or equal to 15. 

cmm - a fifteen element integer*4 array containing the GOLD’S 
CMM outputs, each element in this array has the value 
of either a 0 or a 1. 

all_cmm - an N_ATS by fifteen element integer*4 array containing 
each of the ATis’ CMM outputs, each element in this array 
has the value of either a 0 or a 1. 

record formats: 

first record: rem_n_diff (’ ’, 3i3) 
second record: cmm (’ ’, 15(il,’,’)) 
third record: all_cmm (’ ’, 45(il,’,’)) 
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iABEND.DAT: 


purpose: 

The iABEND.DAT files contain a history of the abend failures 
which have been documented for each ATi during the execution 
the Launch Interceptor Code. 

reference: 

the IABEND.DAT files are referenced individually with unit number 30 
in the following type of open statement: 

open( unit=20, file= iABEND.DAT, access=’ SEQUENTIAL’, status=’OLD’ ) 


data structures: 

function - a character string of length six. this variable contains the 
the name of the module in which the abend occures. 

position - a character string of length two. this variable contains the 
where in the module the abend occurred. 

ele(l) - a character string of length three, this variable contains the 

CMS element with the source for the particular module under 
consideration. 

gen(I) - a byte variable, the generation of the CMS element with the fix 
required to fix the abend in the given module. 

record format: 

function position ele(l) gen(l) 

-or- (tl,a6,t9,a2,tl3,a3,tl8,i2) 
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iCLASS.DAT: 


purpose: 

the records of iCLASS.DAT indicate to the spawnned FEXAPP.COM which 
CMS elements are to be superseded with new generations, i.e. which 
fix to install. 

reference: 

the iCLASS.DAT files are referenced through the N_ATS integer*4 data 
array AT_UNIT. each ATi’s iCLASS.DAT file is assigned a unit number 
which is stored in the ith element of the AT_UNIT array, currently, 
AT_UNIT(1)=11, AT_UNIT( 2)= 12, AT_UNIT(3)=13. the iCLASS.DAT files 
are made available through the statement: 

open) unit=AT_UNIT(i), file=iCLASS.DAT, access=’SEQUENTIAL’, 

+ status=’OLD’ ) 

data structures: 

ele_gen - character* 10, the CMS element name and generation which 
is to be or was superseded in the CMS class for the ATi 
presently under consideration. 

record format: 

ele_gen -or- (lx, lal, li‘2.2, \for(’, lil, ’)’) 



iCLOBBER.DAT: 


purpose: 

The iCLOBBER.DAT files contain a history of the overwrite failures 
which have been doucmented for each ATi during the execution of the 
Launch Interceptor Code. 

reference: 

iCLOBBER.DAT files are referenced individually with unit number 20 
in the following type of open statement: 

open( unit=20, file=iCLOBBER.DAT, access='SEQUENTIAL’, status='OLD’ ) 
data structures: 

clob - integer**!, the clobber value indicating which common region 
was overwritten 

ele(l) - character*3, the CMS element containing this section of 

code with the common region in which the overwrite occurred 

gen(l) - byte, the generation of CMS element from above with the fix 
to stop the overwrite from occurring again 

record format: 

clob ele(l) gen(l) 

- or - (tl,il,t4,a3,t9,il) 
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iCMM.DAT: 


purpose: 

The iCMM.DAT files contain a history of failures and associated fixes 
in the conditions met matrix which have been recorded for each ATi 
during the execution of the Launch Interceptor Code. 

reference: 

iCMM.DAT files are referenced individually with unit number 40 
in the following type of open statement: 

open( unit=40, file=iCMM.DAT, access=’SEQUENTIAL’, 

+ status=’OLD’, recl=24 ) 

data structures: 

ele(REC_CMM) - a REC_CMM element character*3 array, the CMS elements 
which contain the section of code in which it is 
possible for this cmm to fail 

gen(REC_CMM) - a REC_CMM byte array, the generation of the CMS elements 
from above with potential fix(es) for the failed cmm 

nxt(REC_CMM) - a REC_CMM byte array; when searching for a fix, we find 
the element/generation which is currently installed, 
and index off this ’nxt’ field of the same record to 
find the next potential fix, provided another fix exists. 


record formats: 
ele gen nxt 

-or- (t4, a3, t.9, i2, t!3, i2) 
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iLAUNCH.DAT: 


purpose: 

iLAUNCH.DAT files are used when an ATi fails and the failure does 
not show up as an abend, overwrite, or as disagreement in the 
conditions met matrix; a launch error is detected by the assertion 
of a ,not.constraints_met element and the index to the all_cmm array 
being out of bounds. 

reference: 

iLAUNCH.DAT files are referenced individually with unit number 50 
in the following type of open statement 

open(unit=50, file=iLAUNCH.DAT, access=’SEQUENTIAL’, status=’OLD’) 

assumptions: 

only one launch error for any ATi 
data structures: 

ele(l) - a character string of length three, this variable contains the 

CMS element with the source code where the launch error occures 

gen(l) - a byte variable, the generation of the CMS element with the fix 
required to correct the launch error in the given module. 

record format: 
ele(l) gen(l) 

-or- (t3, a.3, t8, i2) 
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SIM. DAT: 


purpose: 

records a history of simulator failures by storing the environmental 
parameters, generated inputs, and simulator and application tasks’ 
outputs. This makes it possible for an operator to determine why the 
simulator failed by examining design stages of previous replications. 

reference: 

SIM.DAT is referenced as unit 1 by all modules in the Autosim, Interface, 
and the Simulator. SIM.DAT is an indexed file; so, unit 1 is accessed with 
the use of keys. The primary key is the sequence number, i.e. the 
sequential number of a record in the file. Modules referencing SIM.DAT 
have read and write privledges to the file. The file is expected to exist 
in the SIMDISK: directory, and an error message will result if the file 
is not found. 

data structures: 

the common regions of SIM.DAT: 

inputs - the generated inputs consist of the following variables: 

X,y,el,r,eps2,a,m,q,epsl,nl,n2,m2,n3,m3,n4, 

m4,n6,bigl,bigr,bige,bign,lcm,pumdia,p,ifout 

outputs - the simulator outputs consist of the following variables: 

cmm, fum, launch, pum 

allouts - the application task outpts consist of the variables: 

all_cmm , all_f u m , all_l au n ch , all_p u m 

all_voterouts - ouput from the voters 

all_v_cmm,all_v_fum,all_vjaunch,all_v_pum, 

all_v_comp_launch 

record format: 

the simulator writes a SIM.DAT record with the following statement, 
the format is implied. 

write(unit=I,iostat=iret)list, inputs, outputs, allouts, all 
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APPENDIX C: LISTING OF AUTOSIM COMMAND PROCEDURES 
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AUTOSIM.COM 


$ assign nlaO: sysSinput 
$ assign out.dat sys$out.put 
$ set default sim_auto_l 
$ set rms/extend=3 
$ cms set library asim_cms_lib 
$ @[sim.probl. auto, tools] bang 
$ set process/name=autosim 
$ run sim_auto_l:autosim 
$ exit 



FORLINREP.COM 


$!!!!!!!!! 

$ dtim = f$time() 

$open/append chart chart. fil 
$ write chart dtim 
$ write chart ” forlinrep.com” 

$ close chart 
$!!!!!!!!!! 

$ delete vtest.map;*,vtest.lis;* 

$ set rms/extend=3 
$ fortran/list/continuations=99 probla 
$ lib/rep simdisk:probllib.olb probla 
$ fortran/list/continuations=99 problb 
$ lib/rep simdisk:probllib.olb problb 
$ fortran/list/continuations=99 problc 
$ lib/rep simdisk:probllib.olb problc 
$ lin/map simdisk:sim, probllib/1, opt/opt - 
/exe=sim_auto_l:sim.exe 
$ delete probl*.obj;* 

$ purge/keep=3 sim.exe,probl*.lis 
$ exit 



SIMBATCH.COM 


$!!!!!!!! 

$ dtim = f$time() 

$ open/append chart chart. fil 
$ write chart dtim 
$ write chart ’’simbatch.com” 

$ write chart ” ” 

$ close chart 

$! 

$ open/append log log.dat 
$ write log ’’start simualtor: ”, dtim 
$ write log ” ” 

$ close log 

$!!!!!!!!!!! 

$ set rms/extend=3 

$ delete sim_auto_l:out.dat;*,sim.rnap;,simbatch.log;,forO*.dat; 
$ delete asim_data:*class.dat;* 

$ purge simdiskiinputs.dat 
$ set process/priority=-5 
$ assign sim_auto_l:simbatch.log forOOG 
$ set process /name=sim 
$ run sim_auto_l:sim 
$ exit 
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VTEST.COM 


$!vtest.com - procedure to test a single version of an AT 
$!!!!!!!!!!!!!!!!!!!!!!!! 

$ open/append log log.dat 
$ open/append chart chart. fil 
$ write log ”vtest” 

$ dtim = f$time() 

$ write chart dtim 
$ write chart ”vtest” 

$ close chart. 

$ close log 
$!!!!!!!!!!!!!!!!!!!!!!!! 

$ set nover 
$ set rms/extend=3 
$ si = ”a” 

$ s‘2 = ”b” 

$ s3 = ”c” 

$ at = s’pl’ 

$ set ver 

$ for/list/cont=99/check=all/obj=tmp probl’at’.for; 

$ lin/map vtest. sub, tmp,simdisk:probllib.olb/l, opt/opt, sim_auto_l:auto.olb/l 
$ set nover 
$ dele tmp.obj;* 

$ open/write tmp_dat: tmp.dat 
$ write tmp_dat: pi 
$ close tmp_dat: 

$ assign/user tmp.dat forSaccept 
$ run vtest 
$ dele tmp.dat;* 

$ dele vtest.exe;* 

$ exit 

$ 
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APPENDIX D. LOG OF AUTOSIM VALIDATION TESTS 



! D.S. Nos. D.S. 


-AT3- 


Fix 

Nos. 



CASE 

ERROR 

0 

ATI: CMM (5) 
AT2: launch error 
AT3: CMM (7) 


123 




AT3: CMM (5,9,10,11, 
12,13,14,15 


ATI: CMM (7) 


ABEND AT3 RADCIR 


AT3: (4,8,13 


ATI: CMM (8,13 


ABEND ATI 
RAD line 30 


203 ATI: CMM (10 


1475 ATI: CMM (5 


AT3 ABEND 


2985 I ABEND ATI 





126 


127 



129 


130 


1 I 1,2 


5 


3 I 3 



AT3: CMM (7 


23 ABEND: ATI 


ATI: CMM (7) 
AT3: CMM (12 


90 AT3: CMM (13 


AT3: CMM (5,9,10,11 



131 

5 

6,9,10,11 

150 




14,15 



I 132 


133 


I 134 


135 


I 136 


137 


138 



0 I 0 



160 ATI: CMM (8,13 


176 ATI: CMM (3 


227 ATI: CMM (5 


892 ABEND: ATI 


T3 

ABEND: AT3 
10000 j End of Rep. 

ATI: overwrite, CMM (5 


AT3: (5,9,10,11,12 


ATI: CMM (7) 


ABEND: ATI 


AT3: CMM (7 


AT3: CMM (4,8,13 


6 1 1 

i 


!| ‘ 

1 1 

5,8,13 il 279 | ATI: CMM (8,13 



!! 370 j ATI: CMM (3 


II 539 i ATI: CMM (5 


i! 568 ABEND: AT3 


ABEND: AT3 


4968 I ABEND: ATI 


1 i 

8 :! 1 ! l 1 

1 

! 10000 1 End of Re 











































































Seq. No. 
314 


-ATI- I -AT2- I 


Fix I Fix 

D.S. Nos. I D.S. Nos. D.S. 

o ~ o To 0 0 


2 2 


3 5 


-AT3- 


Fix 

Nos. 


0 


Hi 


6 6 


340 


I 341 


8 


0 0 


5,6,8,9,10 

11,13,14,15 


6 


6,9,10,11, 


5,8,13 


18 


5,6,8,9,10 

11,12,13, 


ERROR 


ATI: CMM (5) 
AT2: launch error 


ABEND: ATI 


ATI: CMM (7 


ATI: CMM (10) 
AT3: CMM (12,13 


53 AT3: CMM (7 


AT3: CMM (4,5,8,9,10 
11,13,14,15 


ATI: CMM (8,13) 


I 318 ATI: CMM (5 


3970 I ABEND: AT3 


4159 ATI: CMM (10) 
ABEND: AT3 


II 4159 t AT3: CMM (10 


II 6605 ABEND: ATI 


II 10000 End of 


ATI: overwrite, CMM (5) 
0 AT2: launch error 
AT3: CMM (7) 


II 2 | ATI: CMM (7 


AT3: CMM (12,13 


ABEND: ATI 


110 ATI: CMM (8,13) 
AT3: CMM (7) 


AT3: CMM (5,9,10,11 


AT3: CMM (4,8,13) 


I 149 I ATI: CMM (3 


II 237 I ABEND: AT3 


II 1077 ATI: CMM (5 


ABEND: ATI 


I 10000 I End of R 


ATI: overwrite 
0 AT2: launch error 
AT3: CMM (7) 


II 1 ATI: CMM (5 


ATI: CMM (5) 

3 AT3: CMM (4,6,8,9,10, 

11,12,13,14,15 


13 ABEND: ATI 


5 I 3 


23 | ATI: CMM (8,13 


38 I AT3: CMM (7 


316 ATI: CMM (10 


ABEND: AT3 


ABEND: AT3 


5234 ! ABEND: ATI 


10000 | End. of Re 
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